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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
In re application of: 
MICHAEL D- DOYLE et al . 
Application No.: 08/324,443 
Filed: 10/17/94 



For: EMBEDDED PROGRAM OBJECTS IN 
DISTRIBUTED HYPERMEDIA 
SYSTEMS 



Examiner : D . Dinh J£|\j Q £; ]$<)Jj 
Art Unit:, 2317 r ■> - - 
RESPONSE AFTER FINAL 



Assistant Commissioner for Patents 
Washington , D.C. 20231 

Sir: 

The following is responsive to the Office Action mailed 
December 13, 1996: 

REMARKS 

Claims 1-15 and 17-56 are now pending in the 
application. Reexamination and reconsideration are requested. 

Claims 1 and 44 are rejected under 35 U.S.C. §103 as 
being unpatentable over Vetter "Mosaic and the World-Wide Web" 
and further in view of Hansen "Andrew as a Multiparadigm 
Environment for Visual Languages." 

In claim 1, the distributed hypermedia document 
includes an embed text format that specifies the location of an 
object external to the distributed hypermedia document and that 
specifies type information utilized by the browser to identify 
and locate an executable application external to the distributed 
hypermedia document. The browser invokes the executable 
application to display and process the object within the browser 
window . 
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The Vetter reference states that "Users can extend 
Mosaic's functionality by creating custom servers and letting 
other applications control its display remotely." 

The Hansen reference discloses. a multiparadigm language 
environment for visual languages that supports not only graphical 
interaction but also file storage, printing, compilation, and 
execution. It is suggested that each sublanguage element be 
displayed in the same window. 

The examiner states that Vetter discloses that Mosaic's 
functionality can be extended by having custom servers and 
letting other applications control its display remotely. Hence, 
it would have been obvious to extend Mosaic's functionality to 
enable external application to display and process the object 
within the browser- controlled window because it would have 
improved the system by reducing clustering of the display and 
aiding the reader comprehension of the hypermedia document. This 
rejection is respectfully traversed for the following reasons. 

A Rule 131 Declaration is submitted showing that 
applicant's invention was reduced to, practice prior to the 
publication of Vetter. Accordingly, the feature of controlling 
Mosaic's display remotely is not disclosed in the prior art. 

Further, the feature described in Vetter was not 
present in Mosaic prior to the reduction to practice of the 
invention. The feature of remotely controlling Mosaic's display, 
the common -client interface (CCD , was incorporated into Mosaic 
based on the work done by the applicants as evidenced by 
Attachment I , a proposal to the NCSA, the developers of Mosaic, 
to modify Mosaic to allow real-time visualization tools to be 
embedded in Mosaic. (Michael D. Doyle, Ph,D, : Digital Libraries 
Proposal, 01/07/94, page 2 of 6 definition of IRV Server) . 

Accordingly, claims 1 and 44 are deemed allowable over 
the prior art . 

Claims 2-5, 10-14, 24-27, 45-28, and 55 are rejected 
over Vetter in view of Hansen, and further in view of Pilepp et 
al. 

With regard to claim 2, the invention of claim 1 is, 
further limited so that the executable application is a 
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controllable application and includes the step of controlling the 
controllable application from said client workstation via 
communications sent over the distributed hypermedia network. 

The disclosure of Vetter and Hansen is described above. 

Filepp discloses the Prodigy system where a plurality 
of PCs are connected to a host server over the telephone lines 
and run software to receive and transmit data between the host 
and the PCs * 

The examiner states that Vetter disclosed that Mosaic 
and the WWW currently lack direct support for 
application-specific data and support for controlling the 
presentation of nontext data, and suggested letting other 
applications control Mosaic display remotely . This rejection is 
respectfully traversed for the following reasons. 

As described above, the present invention was reduced 
to practice prior to the publication of Vetter and prior to the 
incorporation of the CCI in Mosaic. Accordingly, claim 2 is 
allowable because it depends on claim 1 which is allowable . 

The disclosure of Filepp et al. is unrelated to Mosaic 
or the WWW and merely discloses a host/terminal network 
communication system. 

The rejection of claims 3-5, 10-14, 24-27, 45-48, and 
55 all rely on the teaching of Vetter and the feature of Mosaic 
of letting other applications control its display remotely. 
Because the present invention was reduced to practice prior to 
the publication of Vetter and the incorporation of the CCI in 
Mosaic, these claims are allowable over the prior art. 

In view of the foregoing, Applicants believe all claims 
now pending in this application are in condition for allowance. 
The issuance of a formal Notice of Allowance at an early date is 
respectfully requested . 
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If the Examiner believes a telephone conference would 



expedite prosecution of this application, please telephone the 
undersigned at (41S) 576-0200. 

Respectfully submitted, 



TOWNS END and TOWNS END and CREW LLP 
Two Embarcadero Center, 8th Floor 
San Francisco, California 94111-3834 
(415) 576-0200 
Fax (415) 576-0300 
CEK:db 



ChSrTBB E. Kruege 
Reg. No. 3 0,077 
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NS F/A RP A/NASA Digital Libraries RFP Response 

Title: A Knowledge Managment Environment through the World Wide Web 

Principal Investigator: Michael D. Doyle, Ph.D., UCSF Library and Center for Knowledge Managment 



Specific Aims: 

1 ) To develop a prototype knowledge management environment for the biomedical sciences which 
integrates access to online representations of the scientific literature, bibliographic databases, high- 
performance visualization technologies, large-scale scientific databases, and tools for authoring new- 
generation scientific publications. 

La) To explore and evaluate the applicability of these tools in the areas of radiology and developmental 
& molecular biology. 

2) To provide a means for relating digital forms of spatial, functional, and conceptual information as a 
basis for linking the biomedical scientific literature, through the Red Sage electronic journals project, to 
data resources provided through the Visible Human Project, The Human Brain Project, The Visible 
Embryo Project, The Human Genome Project, The Protein Database, and other large-scale biomolecular 

□ and biostructural databases. 

ffs 

iy 2. a) To exploit these linking strategies in the creation of a set of integrated semi-automatic front ends to 

f!j varied scientific databases accessible through the Internet. 

S 2.b) To incorporate these linking methodologies into interactive authoring and editorial tools, allowing 

~£ the creation of online publications that can embed visualizations and simulations which draw data from 

7*. these Internet-accessible scientific databases. 

3 - ! 

* 3} To develop tools which provide access to interactive visualization and analysis of massive biomedical 
dacascts through the Internet's WorJd Wide Web distributed hypermedia network. 

Mi 3. a) To refine and extend our existing algorithms enabling distributed vi$ualization and analysis 
vj software "engines'* which can be efficiently accessed by remote users through the Internet. 

J* 3.b) To refine and extend our existing algorithms to allow the display and real-time interactive control V 
^ of chree-and four-dimensional data visualization and analysis tools within hypermedia documents 
viewed using NCSA's Mosaic graphical browser to the World Wide Web. 

3.c) To develop algorithms which use novel compression technologies for the optimized interactive 
remote control of computationally-intensive graphical applications through the Internet. 

3.d) To integrate a p b & c into a system which allows real-time remote access to distributed parallel 
computational applications for visualization and analysis resources within a distributed hypermedia 
environment 



4) To explore extensions of the paradigm of scientific publishing which are made possible through use of 
current multimedia technologies in a networked environment, including: 

4. a) publishing multidimensional datasets integrated with articles, eg: MRI and molecular data, 
perferred views, animations, interactive visualizations, interactive mathematical models, and 

4.b) development of scientific authoring tools for publications which exist only in the networked 
environment. 



/ 
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4.b.l) This will include integration of HTML+ WYSIWYG authorial and editorial tools, 
multidimensional data visualization applications, molecular modelling and database management cools 
into an interactive scientific publishing environment. 



System Diagram 




■ss-sf 





» Definitions: 



HTML+: Hypertext Mark-up Language — This is the language that 

World Wide Web databases are encoded In, and that 
Mosaic interprets. 

IRV Server; tfCSF CRM's Interactive Remote Visualization Server » 

This allows interactive real-time visualization tools to be 
embedded into Mosaic documents. 

VLs: UCSF CKM's distributed remote volume visualization tool 

Midas: UCSF CGL's molecular visualization package 

SVM: Sequence Visualization Module - An as-yet unamed tool 

for graphical display of genetic sequence data. 

ITP: Informal Text Phrase — A user-entered search term, or a wqrd or phrase 

of text that the user highlights from within a document, 

URN: Universal Resource Name — A persistent, location- 

Independent identifier for an object 

URL: Universal Resource Location - The address or an object- It 

contains enough information to identify a communications 
protocol and retrieve the objecL 

URC; Universal Resource Characteristics — Any combination of 

one or more URNs or URLs witjfmela~ftjformation (e.g. 
author, format, compression method 
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Description: 

The system wj|J draw from of a number of fundamental databases including bibliographic data (Medline) in 
the form of MARC records, journal publication data (Red Sage) in the form of SGML header and 
Postscript files, encyclbpcdic reference text data (CMDT) scored in an object-oriented SGML database, 
volumetric anatomical data (Visible Human Project) stored as NCSA HDFdatasets, protein structure data 
(Protein Data Bank) stored as PDB files, and genetic sequence data (GenbanJc) stored as compressed ASCII 
strings (7, I'm guessing about Genbank). 

These databases will reside behind a Z39.50 interface layer which yeilds, to the requesting client, the 
respective dataset in its native form. This data then goes through a translation layer where the data is either 
translated directly into HTML+ (Medline. Red Sage, CMDT) or loaded into a native^data visualization tool 
(Visible Human, PDB. Genbank). The HTML+ code is then passed to a set of HTML+ servers, which can 
be browsed by the Mosaic client. The visualization data is handled differently. The graphical I/O of the 
relevant visualization tool is passed to an interactive remote visualization (IRV) server* which handles both 
mapping of the display output from the visualization tool onto embedded live-visualization windows within 
the Mosaic-browsable HTML+ documents, as well as capture of user-entered mouse and keyboard events 
within the visualization windows and transmission of those mouse and keyboard events back to the relevant 
visualization tools. The user, browsing the system with the project's enhanced version of the Mosaic client, 
is presented with data and visualizations derived from these various databases, yet embedded into coherent, 
Q multimedia Mosaic documents. 

5 .j For multimedia documents that have been explicitly pre-composcd, the linking of these various data 

^ resources can cake the form of universal resource names (URNs) that are encoded as tags into the HTML+ 

" J documents. This is passed to the system's semantic domain name server, for resolution of the information 

^ object's location and retrieval means. The URNs are used as indices in order to look up the relevant 

# universal resource characteristics (URCs) in a URC database, which yeilds the universal resource location 

*S (URL), or physical adress, of the information object in question. 

a s 

3 Semi-automatic means will be provided for a user to search for arbitrary information objects on the system 
La, by either keying in a search word or phrase, or by highlighting a not-already-hyperlinked section of text that 
(s)he happens to be viewing within the Mosaic client at the time. This informal text phrase (ITP) is then 
passed to the semantic domain name server, which passes it on to a universal resource thesaurus (which will 
f ™ incorporate elements of the NLM's UMLS system). The thesaurus compares the ITP to its database of 
j terms and phrases and returns a rank-ordered list of URNs that are likely to match the object in question. 
*3j These URNs arc then passed to the URC database for resolution of URLs that point to information objects 
«P on the Internet that are most likely to match the ITP that the user employed to initiate the search. The user 
is presented with a rank ordered set of textual descriptions of likely matches which are hyperlinked, via 
their URLs, to the data in question. Clicking upon a selection from this list loads the related data into the 
relevant visualization server (IRV) or HTML+ server, and a second Mosaic window pops up to allow 
viewing or interaction with that dataset. 

A set of authoring and editing tools will be designed to allow the interactive WYSIWYG creation of 
HTML* documents, as well as allowing the embedding of visualizations, etc., which.can be created using 
the interactive remote visualization tools, and which can use data from the various scientific databases 
mentioned above. Alternatively, the author can use his/her own datasecs, which would be uploaded to an 
Internet-accesible World Wide Web server. The journal editor can use the same set of tools to edit 
submitted articles and to communicate changes to the text with the author. This* of course, would occur in a 
private, access -control led, area of trie system, so that confidentiality of the material to be published can be 
controlled. 

Other private, access-controlled HTML+ servers will be used to administer the peer review process. A 
modification of NCSA's Mosaic-based group annotation server will be developed to allow the jounal editor 
to exercise precise control and documentation of each reviewer's comments and suggestions. 
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Contributions: 
UCSFCKM; 



« Development of Z39-50-cornpliant experimental (subset) databases for storage of Visible 
Human data, PDB data, and Genbank sequence data. 

• Cooperation with AT&T in the development of an object-oriented SGML-based database fqr 
the Handbook of Current Medical Diagnosis and Treatment (CMDT) 

• Development of an experimental 239.50 interface to Medline data ( will be unnecessary if UCs 
DLA can provide such an interface to Melvyl Medline early enough into the project timeline) 

• Development of translator servers to translate Medline MARC records, CMDT SGML data and 
Red Sage SGMIJPostscript data into HTML+ 

• Development of a set of HTML+ documents that act as browsers to Medline, CMDT, and Red 
Sage 

• Refinement and further development of Vis to allow better distribution of computation and 
better integration with Mosaic, 

• Cooperation with CGL to adapt Midas for integration within Mosaic, and to identify and adapt 
a suitable program for graphical display of genetic sequence data. 

• Refinement and further development of the interactive remote visualization server, and its 
incorporation (with NCSA's help) within the Mosaic environment. 

• Development, in cooperation with NCSA, of an enhanced version of the Mosaic client to allow 
easier integration of external programs within Mosaic-readable documents. 

• Development, in cooperation with Springer- Veriag and NCSA, of an interactive WYSIWYG 
editor for creation of HTML+ documents, and for embedding visualizations created using 
CKM's IRV tools, as well as development of a modified version of NCSA's group annotation 
server to support the peer review process. 

• Development, in cooperation with AT&T, of an object-oriented SGML-based LTRC database 

• Development, in cooperation with UCSFs CGL, UCSFs Radiology DepL, Washington Univ.. 
and AT&T, of a Semantic domain name Server and a URN Thcsarus, based upon AT&T's 
object-oriented SGML database technology. 

• Development, in cooperation with UCSFs CGL, UCSFs Radiology Dept., Washington 
Univ., and Springer- Veriag of a set of sample content for use in evaluating the effectiveness of 
the system, as well as for demonstration of the results of the project. 



UCSF CGL: 



• Cooperation with CKM to adapt Midas for integration within Mosaic, and to identify and adapt 
a suitable program for graphical display of genetic sequence data. 



• Contributing to the refinement and further development of the interactive remote visualization 
server, and its incorporation (with NCSA's help) within the Mosaic environment. 
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. Development, in cooperation with UCSFs CKM, UCSFs Radiology Dept.. Washington Univ., 
and AT&T, of a Semantic domain name server and a URN Thesarus, based upon AT&Ts 
object-oriented SGML database technology. 

• Development, in cooperation with UCSFs CKM, UCSFs Radiology Dept. Washington 
Univ., and Springer- Verlag of a sec of sample content for use in evaluating the effectiveness of 
the system, as well as for demonstration of the results of the project. 

Washington University: 

. Development, in cooperation with UCSFs CKM. UCSFs CGL, and AT&T, of a Semantic 
domain name server and a URN Thesarus, based upon AT&Ts object-oriented SGML database 
technology. 

. Development, in cooperation with UCSFs CKM, and UCSFs CGL., and Springer Verlag of a 
set of sample content for use in evaluating the effectiveness of the system, as well as for 
demonstration of the results of the project. 

AT&T Bell Laboratories: 

■ Development of Z39,50 interface to the RightPages server.. 

• Cooperation with CKM in the development of an object-oriented SGML-based database for 
the Handbook of Current Medical Diagnosis and Treatment (CMDT) 

• Development, in cooperation with CKM. of an object-oriented SGML-based URC database 

• Development, in cooperation with UCSFs CGL, UCSFs Radiology DepL, Washington Univ.. 
and CKM. of a Semantic domain name server and a URN Thesarus, based upon AT&Ts 
object-oriented SGML database technology. 

Springer-Verlag: 

• Development, in cooperation with UCSFs CKM and NCSA, of an interactive WYSIWYG 
editor for creation of HTML+ documents, and for embedding visualizations created using 
CKM's IRV tools, as well as development of a modified version of NCSA's group annotation 
server to support the peer review process. 

• Development, in cooperation with UCSFs CKM, and UCSFs CGL., and Washington Univ. of a 
set of sample content for use in evaluating the effectiveness of the system, as well as for 
demonstration of the results of the project. 



F% E 




NCSA: 



• Cooperation with CKM jn developing an enhanced version of Mosaic to allow easier integration 
of a client modeule for CKM's interactive remote visualization server. 



• Cooperation with CKM and Springer- Verlag in the modification of NCSA's group annotation 
server to facilitate the peer-review process. 
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Personnel: 
Co-Investigators: 
UCSF: 

Library & CKM: Richard Lucier, David Martin, Zoc Ssavn. Ph.D.. Cheong Ang, Marc 

Salomon 
Radiology: Tom Budinger, Ph.D. 

Molecular & developmental Biology: Tom Ferrin, Ph.D.. Charles Ordahl, PhD. 

Washington University (molecular biology): Toni Kazic, PhD 

Bell Laboratories: Ed Szurkowski. Guy Story 

Springer Verlag: Bob Badger, PhD 

NCSA; Joseph Hardin, PhD, & Mosaic development group 

SFSU: Computer Science Dept. MS students 

Timetable: 4 years 

Budget; 1.2 $M/year 




Figure 1: A siareo-palr Illustration of Interactive real-time 3-dlmenslonal human embryonic volume 
reconstructions embedded within an NCSA Mosaic document This technology was dovoloped by iha Cenier for 
Knowledge Management at the University of California, San Francisco, and was demonstrated there In 
November, 1993. 
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Major work experience 

University of Illinois, 

Associate Directory Software Development Group, NCSA, 1992-present 
University of Illinois, 

Manager, Software Development Group, NCSA, 1988-1992 
Coordinator, Academic Affiliates Program, NCSA, 19S7 -198S 
Visiting Research Associate, NCSA, 1986-1987 

University of Georgia at Athens, 

Visiting Instructor, Department of Speech Communication, 1985-1986 

University of Illinois, 
® Teaching Assistant, Department of Speech Communication, 1979-1985 
LJ Teaching Assistant; Department of English-Rhetoric, 1978 
fy Research Assistant, Department of Sociology, 1977-1978 

Other work 

JG Consultant -Business computer communications, database management and office 
hi organization, 1982-1985 

7 Consulting Editor/Contributor-The Champaign-Urbana Weekly news magazine, 1981-1982 
U Editor-The Champaign-Urbana Weekly news magazine, 1979-1980 

Mi Recent Grants and Awards 

^ Exploratory Research and Initial Development of Software for the Analysis of Multiple 
=Q Hybridization Images, 1990-1992 

£: IBM Second Generation RISL Workstation Graphics Capabilities: Jointly Defined Effort 
1990-1992 

NCSA Hierarchical Data Pile Software Capitalization: National Distribution and 

Support, 199M993 
Research Education for Undergraduate Students Supplement 1990-1993 
Supercomputers for Biologists: Macromolecular Sequence Analysis through Distributed 

Computing, 199M993 

Visualization in the MS DOS Environment' NCSA/Jackson State Collaborative Project, 1990- 
1993 

x3d Program Development 1?°2-1993 
Education 

Study toward Ph-D., Speech Communication, University of Illinois 
Study Abroad, University of Cologne, Cologne, West Germany, 1978 
B.A., History, University of Illinois, 1972 
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Selected Recent Conference Papers/Participation 

Sigchi, Session Paper, "Scientific Visualization in a Collaborative Environment" Amsterdam, 

Holland, April, 1993 (in submission} 
Sigchi, Session Paper, "Collaborative Hypermedia for Computational Science/ Amsterdam, 

Holland, April, 1993 (in submission) 
Oceanographic Institute, Old Dominion Univer3ity / Invited Seminar, "Recent Developments in 

Collaboration Technologies;' Norfolk, Va., Feb. 1993 
Scientific Computing & Automation Conference, Invited Speaker, "Cro$s platform Digital 

Conferencing Software Development," Washington, DC 1992 
New York University, Academic Computing Facility, "Scientific Visualization in Collaborative 

Technologies," New York City, NY, 1992 
International Institute of Ecological Economics Conference, Invited Workshop Participant, Beijer 

Institute, Stockholm, Sweden, 1992 
American Educational Research Association Conference, "Collaborative Tools for Scientific 

Communication and Understanding," San Francisco, CA, 1992 
Mac SciTech National Conference'92, "Interpersonal Computing for Computational Scientists: The 

NCSA Collage Series/ 1 San Francisco, CA, 1992 



